Papers with structural probing
Introducing Orthogonal Constraint in Structural Probes (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on interpreting pre-trained models' representations and analyzing their structures. |
| Approach: | They propose a new type of structural probing where a linear projection is decomposed into two types. |
| Outcome: | The proposed method is tested on two novel tasks and shows that lexical and syntactic information is separated in the representations. |
Syntax-guided Contrastive Learning for Pre-trained Language Model (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies rely on additional syntax-driven attention components to enhance the transformer, which require more parameters and additional syntactic parsing in downstream tasks. |
| Approach: | They propose a syntax-guided contrastive learning method which does not change the transformer architecture and does not alter the transformer structure. |
| Outcome: | The proposed method achieves consistent improvements in a variety of tasks including grammatical error detection, entity tasks, structural probing and GLUE. |